The Coalescent Process With Selfing Magnus Nordborg ” and Peter Donnelly ’

A method for estimating the selfing rate using DNA sequence data was recently proposed by Milligan. Unfortunately, a number of errors make interpretation of his results problematic. In the present paper we first show how the usual coalescent process can be adapted to models that include selfing, and then use this result to find moment estimators as well as the likelihood surface for the selfing rate, s, and the scaled mutation rate, 0. We conclude that, regardless of the method used, large sample sizes are necessary to estimate s with any degree of certainty, and that the estimate is always highly sensitive to recent changes in the true value. T HERE is long-standing interest in estimating the degree of self-fertilization, or selfing, in partially selfing populations. Several methods have been suggested, most of them based on allozyme frequency data (BROWN 1990). Recently, MILLIGAN ( 1996) suggested a method using DNA sequence data based on the difference in coalescence time for alleles sampled within and between an individual. In this article, we first show in general how the standard neutral coalescent process can be used for models with partial selfing. We use this result to derive alternatives to MILLIGAN’S estimator and to compare the properties of the estimators. Finally, we discuss the usefulness of DNA data for estimating the selfing rate. THE COALESCENT PROCESS WITH PARTIAL SELFING Consider a Wright-Fisher model of N diploid hermaphrodites, with Nvery large. Define the selfing rate s as the fraction of offspring that is produced by selffertilization. The remaining fraction 1 s is produced via random mating. For a single locus with two alleles, it is easy to show that the genotypic proportions quickly converge to p‘ + pqF, 2pq( 1 F) , and q‘ + pqF, where